Using data bases to test methods for decisions under uncertainty

To address the need for efficient and unbiased experimental testing of methods for decision under uncertainty, we devise an approach for probing weaknesses of these methods by running numerical experiments on readily available or easily obtainable databases of real life data. Since the approach uses real life data, it allows us to study the effect of modeling error on the performance of a method. For illustration, we apply probabilistic and possibilistic approaches to a database of results of a domino tower competition. The experiments yielded several surprising results. First, even though a probabilistic metric of success was used, there was no significant difference between the rates of success of the probabilistic and possibilistic models. Second, the common practice of inflating uncertainty when there is little data about the uncertain variables shifted the decision differently for the probabilistic and possibilistic models, with the latter being counter-intuitive. Finally, inflation of uncertainty proved detrimental even when very little data was available.